Model forming apparatus, model forming method, photographing apparatus and photographing method

ABSTRACT

The present invention provides a model forming apparatus that can simply and efficiently form a three-dimensional model of an object using previously obtained three-dimensional model data of the object as a starting point. The apparatus comprises a photographing section  110  for photographing an object  10 , an image data storage section  130  for storing three-dimensional model data of the object, a display section  140  for displaying a three-dimensional model based on the three-dimensional model data of the object  10 , a recognition section  150  for recognizing an unmodeled part of the object  10  based on the three-dimensional model data stored in the image data storage section  130 , and a photographing instruction information section  160  for obtaining photographing instruction information related to photographing the unmodeled part. The photographing section  110  photographs the object  10  in accordance with the photographing instruction information obtained by the photographing instruction information section  160.

FIELD OF THE INVENTION

The present invention relates to a model forming apparatus and a modelforming method that can provide a photographer with photographingconditions, such as the photographing position, photographing direction,based on given conditions of an object, such as the image resolvingpower, positional accuracy, area of its unmodeled part, to form athree-dimensional model of the object. The present invention alsorelates to a photographing apparatus and a photographing method suitableto efficiently generate a 3D (three-dimensional) model of an object fromimages of the object photographed from the air or on the ground.

BACKGROUND OF THE INVENTION

Recent development in digital technology has made it possible to easilyobtain detailed 3D data and high-resolution image data on a measuringobject using a laser scanner or a digital photogrammetric technique. Forexample, a side surface of an object can be precisely measured from theground, using close-range photogrammetry with a digital camera or usinga ground-based measuring instrument.

However three-dimensional data obtained with a measuring instrument thatmeasures the position of, or the distance and direction to, ameasurement point (hereinafter simply referred to as “measuringinstrument”), such as a laser scanner, an auto-tracking measuringinstrument or a non-prism measuring instrument are basically composed ofthree-dimensional coordinate data including distance data. Therefore, ithas become apparent that it is difficult to correlate the 3D dataobtained with a measuring instrument with the site conditions, and henceto determine which part of a measuring object has been measured in tyingthe 3D data to image information of the measuring object, resulting inthe complicated formation of a three-dimensional model of the object.Also, when 3D data are obtained with a measuring instrument installed onthe ground, 3D data from the air, for example, 3D data on the top of abuilding or a road from the air, cannot be obtained to make it difficultto form a three-dimensional model of an object.

On the other hand, digital cameras and laser scanners for installationon flying bodies, such as helicopters or airplanes, have been developedin recent years. When images photographed from the air are combined withthose photographed on the ground to generate a 3D model of an object,however, a significant amount of image distortion can occur due to thedifference in photographing direction, photographing scale or lensaberration, causing a load of image corrections and hence making itdifficult to form a three-dimensional model of the object.

The present invention has been made to solve the foregoing problems, andtherefore has an objective to provide a model forming apparatus and amodel forming method that can simply and efficiently form athree-dimensional model of an object using previously obtainedthree-dimensional model data of the object as a starting point.

Solving the foregoing problems, the present invention has anotherobjective to provide a photographing apparatus and a photographingmethod that can efficiently generate a 3D model of an object bycorrecting images of the object photographed from the air or on theground.

SUMMARY OF THE INVENTION

In order to achieve the above objectives, a model forming apparatusaccording to the present invention comprises, as shown in FIG. 1 forexample, a photographing section 110 for photographing an object 10; animage data storage section 130 for storing three-dimensional model dataof the object 10; a display section 140 for displaying athree-dimensional model based on the three-dimensional model data of theobject 10; a recognition section 150 for recognizing an unmodeled partof the object 10 based on the three-dimensional model data stored in theimage data storage section 130; and a photographing instructioninformation section 160 for obtaining photographing instructioninformation related to photographing the unmodeled part, wherein thephotographing section 110 photographs the object 10 in accordance withthe photographing instruction information obtained by the photographinginstruction information section 160.

In the apparatus configured as above, the recognition section 150recognizes a missing photographing direction for the object, thephotographing instruction information section 160 obtains photographinginstruction information necessary to photograph the object so as to makeup for the missing photographing direction, and the display section 140displays the information that allows the photographer to recognize theinformation. Thus, the apparatus suitably allows the photographer tophotograph the object from the missing photographing direction tosupplement the image data in an appropriate manner.

In order to achieve the above objectives, a model forming apparatusaccording to the present invention comprises, as shown in FIG. 19 forexample, a photographing section 110 for photographing an object 10; animage data storage section 130 for storing at least three-dimensionalmodel data of the object 10 or object image data obtained byphotographing the object with the photographing section 110; a modelforming section 155 for obtaining three-dimensional model data of theobject 10 from the object image data stored in the image data storagesection 130; a display section 140 for displaying a three-dimensionalmodel based on the three-dimensional model data of the object 10; arecognition section 150 for recognizing an unmodeled part based on thethree-dimensional model data stored in the image data storage section130; and a photographing instruction information section 160 forobtaining photographing instruction information related to photographingthe unmodeled part, wherein the display section 140 displays thethree-dimensional model and the photographing instruction informationtogether.

In the apparatus configured as above, the recognition section 150recognizes a missing photographing direction for the object, thephotographing instruction information section 160 obtains photographinginstruction information necessary to photograph the object so as to makeup for the missing photographing direction, and the display section 140displays the information that allows the photographer to recognize theinformation. Thus, the apparatus suitably allows the photographer tophotograph the object from the missing photographing direction tosupplement the image data in an appropriate manner. The model formingsection 155 obtains model data of the object 10 from the object imagedata stored in the image data storage section 130. The display section140 displays a three-dimensional model based on the three-dimensionalmodel data of the object 10, allowing the photographer to recognize anappropriate supplemental photographing direction for the object, inaddition to the unmodeled part recognized by the recognition section150. Thus, the photographer can supplement the image data byphotographing the object from the missing, or an appropriatesupplemental, photographing direction in an appropriate manner.

The model forming apparatus according to the present invention mayfurther comprise, as shown in FIG. 19, a texture attaching section 192for dividing the model formed from the three-dimensional model data bythe model forming section 155 into plural areas and attaching texture ofthe object image data to each divided area based on the relationshipbetween the each divided area and a photographing direction, wherein thedisplay section 140 displays the model with texture formed by thetexture attaching section 192.

Preferably, in the model forming apparatus according to the presentinvention, as shown in FIGS. 1 and 19, the display section 140 may beconfigured to graphically display the photographing instructioninformation obtained by the photographing instruction informationsection 160 as superimposed over a plan view of an area containing, orover a stereo model of, the object 10.

Preferably, the model forming apparatus according to the presentinvention may further comprise, as shown in FIGS. 1 and 19, ameasurement setting section 170 for setting necessary measurementconditions (measurement accuracy, baseline lengths and overlap); and aphotographing condition calculation section 180 for obtainingphotographing conditions satisfying the measurement conditions set bythe measurement setting section 170.

In order to achieve the above objectives, a model forming method causes,as shown in FIG. 2 for example, a computer to perform the steps of:displaying a three-dimensional model of an object 10 based onthree-dimensional model data of the object 10 stored in an image datastorage section 130 (S110); recognizing an unmodeled part of the object10 based on the three-dimensional model data (S120); and obtainingphotographing instruction information related to photographing theunmodeled part (S130). And the photographer photographs the object 10with the photographing section 110 in accordance with the photographinginstruction information (S140).

In order to achieve the above objectives, a model forming method causes,as shown in FIG. 20 for example, a computer to perform the steps of:obtaining model data of an object 10 from three-dimensional model dataof the object 10 stored in an image data storage section 130 or fromobject image data obtained by photographing the object with aphotographing section 110 (S210); displaying a three-dimensional modelbased on the three-dimensional model data of the object 10 (S220);recognizing an unmodeled part of the object 10 based on thethree-dimensional model data of the object 10 (S230); obtainingphotographing instruction information related to photographing theunmodeled part (S240); and displaying the three-dimensional model andthe photographing instruction information together in a display section140 (S250). The object 10 is photographed by the photographer with thephotographing section 110 in accordance with the three-dimensional modeland the photographing instruction information displayed in the displaysection 140 (S260).

In order to achieve the above objectives, a photographing apparatuscomprises, as shown in FIG. 29 for example, a photographing section 110for photographing an object 10; a photographing position measurementsection 120 for obtaining photographing position information of thephotographing section 110; an image data storage section 130 for storingplural image data of the object 10 with a known positional point; amodel forming section 155 for forming a three-dimensional model of theobject 10 using the image data stored in the image data storage section130; and a model display section 144 for displaying, for thethree-dimensional model of the object 10 formed by the model formingsection 155, a three-dimensional model image of the object 10 as viewedfrom a photographing position of the photographing section 110 based onthe image data stored in the image data storage section 130 and thephotographing position information obtained by the photographingposition measurement section 120.

In the apparatus thus configured, the model display section 144 displaysa three-dimensional model image of the object 10 as viewed from aphotographing position of the photographing section 110 when thephotographing section 110 has moved. This allows the photographer todetermine whether or not a particular photographing position of thephotographing section 110 is an appropriate photographing positionnecessary to form a model of the object 10.

Preferably, in the photographing apparatus according to the presentinvention, the model forming section 155 may form a three-dimensionalmodel of the object 10, using a first photographed image stored in theimage data storage section 130 together with the photographing positioninformation and a second photographed image displayed in the modeldisplay section 144, and the apparatus may further comprise: aphotographing condition measurement section 220 for obtaining at leastone of measurement accuracy, baseline, photographing position,photographing angle, and interval between photographing sections forthree-dimensional measurement on the object 10 from the photographingposition information related to the first and second photographedimages.

In order to achieve the above objectives, a photographing apparatuscomprises, as shown in FIG. 31 for example, a photographing section 110for photographing an object 10; a photographing position designatingsection 280 for allowing designation of a photographing position of thephotographing section 110; an image data storage section 130 for storingplural image data of the object 10 with a known positional point; amodel forming section 155 for forming a three-dimensional model of theobject 10 using the image data stored in the image data storage section130; and a model display section 165 for displaying, for thethree-dimensional model of the object 10 formed by the model formingsection 155, a three-dimensional model image of the object 10 as viewedfrom the designated photographing position of the photographing section110 based on the image data stored in the image data storage section 130and the photographing position designated by the photographing positiondesignating section 280.

In the apparatus configured as above, the model display section 165displays a three-dimensional model image of the object 10 as viewed fromthe photographing section 110 when the photographing section 110 hasmoved to a photographing position designated by the photographingposition designating section 280. This allows the photographer todetermine whether or not the photographing position of the photographingsection 110 designated by the photographing position designating section280 is an appropriate photographing position necessary to form a modelof the object 10.

Preferably, the photographing apparatus according to the presentinvention may further comprise: a photographing position measurementsection 120 for obtaining photographing position information of thephotographing section 10; and a photographing condition measurementsection 220 for obtaining at least one of measurement accuracy,baseline, photographing position, photographing angle, and intervalbetween photographing sections for three-dimensional measurement on theobject 10 from the photographing position information related to thefirst and second photographed images. In this event, the first andsecond photographed images are such that the model forming section 155forms a three-dimensional model of the object 10 using a firstphotographed image stored in the image data storage section 130 togetherwith the photographing position information and a second photographedimage displayed in the model display section 165.

Preferably, in the photographing apparatus according to the presentinvention, the image data storage section 130 stores at least a set ofstereo images photographed with a photographing device other than thephotographing section 110 that can be used to form a three-dimensionalmodel of the object 10 in the model forming section 155. The set ofstereo images may include, for example, a stereo image (small-scaleaerial photograph) photographed with a photographing device other thanthe photographing section 110. Usually, the stereo images arephotographed with a photographing device other than the photographingsection 110 and preferably wide-area photographed images compared to theimage photographed with the photographing section 110, and refer tosmall-scale, low-resolution and wide-area photographed images. As analternative to an image photographed with other than the photographingsection of the present photographing apparatus, solely an imagephotographed with the photographing section 110 of the presentphotographing apparatus may be used to perform, for example, the processof forming a 3D model.

In the photographing apparatus of the present invention, the image datastored in the image data storage section 130 may preferably include atleast one of a stereo aerial photograph, a stereo small-scalephotograph, a stereo low-resolution image and a wide-area photographedimage, photographed with a photographing device other than thephotographing section 110.

Preferably, in the photographing apparatus according to the presentinvention, the image data obtained by photographing with thephotographing section 110 may be stored in the image data storagesection 130 such that the known positional point of the object 10 can bedisplayed being superimposed over the object image; and the object imagestored in the image data storage section 130 can therefore be used toform a three-dimensional model of the object 10 in the model formingsection 155.

Preferably, in the photographing apparatus according to the presentinvention, the photographing section 110 may photograph a firstphotographed image to be stored in the image data storage section 130together with the photographing position information and a secondphotographed image photographed with having an overlapping area with thefirst photographed image with respect to the object 10; and the imagedata storage section 130 may be configured to sequentially store thefirst photographed image and the second photographed image.

Preferably, in the photographing apparatus according to the presentinvention, as shown in FIGS. 29 and 31 for example, the photographingsection 110 may photograph the object 10 within view sequentially fromdifferent photographing positions, and the photographing apparatus mayfurther comprise a monitor image display section 142 for displaying anobject image based on image data of the object 10 obtained bysequentially photographing the object 10.

Preferably, the photographing apparatus according to the presentinvention may, as shown in FIG. 29 for example, further comprise areference point display position computing section 190 for displaying,in the monitor image display section 142, at least one of a referencepoint or a pass point superimposed over the image data obtained bysequentially photographing the object 10.

Preferably, the photographing apparatus according to the presentinvention may, as shown in FIGS. 29 and 31 for example, furthercomprises a measurement condition display section 146 for displayingphotographing conditions obtained by measurement with the photographingcondition measurement section 220.

In order to achieve the above objectives, a photographing apparatus 300according to the present invention comprises, as shown in FIG. 33 forexample, a photographing section 110 for photographing an object 10; afinder image display section 290 for displaying the object image beingphotographed by the photographing section 110; an image data storagesection 130 for storing plural image data of the object 10 with a knownpositional point; and a data forming section 240 for forming referencepoint data or a three-dimensional model of the object 10 using the imagedata stored in the image data storage section 130, wherein the finderimage display section 290 is configured to display the reference pointdata or the three-dimensional model over the object image.

In the apparatus configured as above, reference point data or athree-dimensional model of the object 10 formed by the data formingsection 240 using the image data stored in the image data storagesection 130 are displayed over an image of the object 10 photographedwith the photographing section 110 and displayed in the finder imagedisplay section 290. This allows the photographer to determine whetheror not a particular photographing position of the photographing section110 has already been used to obtain reference point data or to form athree-dimensional model of the object 10, and makes the photographingwork necessary to form a three-dimensional model of the object 10proceed smoothly.

In order to achieve the above objectives, a photographing methodaccording to the present invention causes, as shown in FIG. 30 forexample, a computer to perform the steps of: obtaining photographingposition information for photographing an object 10 with a photographingsection 110 (S710); forming a three-dimensional model of the object 10using image data stored in an image data storage section 130 for storingplural image data of the object 10 with a known positional point (S720);and displaying, for the three-dimensional model of the object 10, athree-dimensional model image of the object 10 as viewed from aphotographing position of the photographing section 110 based on theimage data stored in the image data storage section 130 and thephotographing position information (S730).

In order to achieve the above objectives, a photographing methodaccording to the present invention causes, as shown in FIG. 32 forexample, a computer to perform the steps of: entering information on aphotographing position designated for photographing an object 10 with aphotographing section 110 (S810); forming a three-dimensional model ofthe object 10 using image data stored in an image data storage section130 for storing plural image data of the object 10 with a knownpositional point (S820); and displaying, for the three-dimensional modelof the object 10, a three-dimensional model image of the object 10 asviewed from the designated photographing position of the photographingsection 110 based on the image data stored in the image data storagesection 130 and the designated photographing position (S830).

According to the model forming apparatus of the present invention, therecognition section recognizes a missing photographing direction for theobject, and the photographing instruction information section allows thephotographer to recognize photographing instruction informationnecessary to photograph the object so as to make up for the missingphotographing direction, allowing the photographer to photograph theobject from the missing photographing direction to supplement the imagedata in an appropriate manner. Then, for example by bundle adjustment ofan aerial photograph and an image photographed on the ground, a 3D modelof the object can be easily constructed.

According to the photographing apparatus of the present invention, themodel display section displays a three-dimensional model image of theobject as viewed from a photographing position of the photographingsection when the photographing section has moved. This allows thephotographer to determine whether or not a particular photographingposition of the photographing section is an appropriate photographingposition necessary to form a model of the object, and makes thephotographing work necessary to form a three-dimensional model of theobject proceed smoothly.

Also, according to the photographing apparatus of the present invention,the model display section displays a three-dimensional model image ofthe object as viewed from a photographing position of the photographingsection when the photographing section has moved to the photographingposition designated by the photographing position designating section.This allows the photographer to determine whether or not thephotographing position of the photographing section designated by thephotographing position designating section is an appropriatephotographing position necessary to form a model of the object, andmakes the photographing work necessary to form a three-dimensional modelof the object proceed smoothly.

Also, according to the photographing apparatus of the present invention,reference point data or a three-dimensional model of the object formedby the data forming section using the image data stored in the imagedata storage section are displayed over an image of the objectphotographed with the photographing section and displayed in the finderimage display section. This allows the photographer to determine whetheror not a particular photographing position of the photographing sectionhas already been used to obtain reference point data or to form athree-dimensional model of the object, and makes the photographing worknecessary to form a three-dimensional model of the object proceedsmoothly.

The basic Japanese Patent Applications No. 2004-203987 filed on Jul. 9,2004 and No. 2004-181809 filed on Jun. 18, 2004 are hereby incorporatedin their entirety by reference into the present application. The presentinvention will become more fully understood from the detaileddescription given hereinbelow. The other applicable fields will becomeapparent with reference to the detailed description given hereinbelow.However, the detailed description and the specific embodiment areillustrated of desired embodiments of the present invention and aredescribed only for the purpose of explanation. Various changes andmodifications will be apparent to those ordinary skilled in the art onthe basis of the detailed description.

The applicant has no intention to give to public any disclosedembodiments. Among the disclosed changes and modifications, those whichmay not literally fall within the scope of the present claimsconstitute, therefore, a part of the present invention in the sense ofdoctrine of equivalents.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the invention (especially in the context of thefollowing claims) are to be construed to cover both the singular and theplural, unless otherwise indicated herein or clearly contradicted bycontext. The use of any and all examples, or exemplary language (e.g.,“such as”) provided herein, is intended merely to better illuminate theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a general block diagram illustrating the functionality of afirst embodiment of the present invention.

FIG. 2 is a flowchart illustrating a first model forming method of thepresent invention.

FIG. 3 shows the system configuration of the first embodiment of thepresent invention.

FIG. 4 is an exemplary screen view displayed on a liquid crystal displaypanel.

FIG. 5 is another exemplary screen view displayed on the liquid crystaldisplay panel.

FIG. 6 illustrates an exemplary photographing position entry screen usedto enter the photographing position.

FIG. 7 is a plan view showing obtained photographing positions.

FIG. 8 is an exemplary 3D model view showing the obtained photographingpositions.

FIG. 9 is an exemplary 3D model view showing the obtained photographingpositions.

FIG. 10 shows exemplary visual images of an object as viewed fromdesignated photographing positions.

FIG. 11 is a flowchart illustrating measurement of a 3D model usingimages photographed from the air and on the ground.

FIG. 12 shows an overall site plan of the object for 3D model generationusing a virtual guiding system according to the present invention.

FIG. 13 is an aerial photograph of about 4 km square around ItabashiWard, Tokyo, including the location where the main office of TOPCONCorporation is registered, provided by the Geographical SurveyInstitute.

FIG. 14 shows a modeled image of an object area for 3D model generationby a model forming section.

FIG. 15 shows images photographed with a photographing section 110 on apowered glider to model the object from the air.

FIG. 16 illustrates photographing from a powered paraglider (paragliderwith an engine).

FIG. 17 illustrates an exemplary process of forming a 3D model using theaerial photograph.

FIG. 18 shows the scene of photographing the object with thephotographing section 110 from an appropriate supplemental direction,and additionally photographed images.

FIG. 19 is a general block diagram illustrating the functionality of asecond embodiment of the present invention.

FIG. 20 is a flowchart illustrating a second model forming method of thepresent invention.

FIG. 21 shows an image of the object in which wireframes are attached toan aerial photograph.

FIG. 22 shows an image of the object in which texture is attached to theaerial photograph.

FIG. 23 illustrates a 3D model with a building adjacent to the objectadded.

FIG. 24 is a flowchart of the recognition of a missing part.

FIG. 25 is a plan view showing missing parts when the object (anarchitecture) is extracted.

FIG. 26 is a 3D model view showing the missing parts when the object (anarchitecture) is extracted.

FIG. 27 is a plan view showing obtained photographing positions.

FIG. 28 is a general diagram illustrating the constitution of a thirdembodiment of the present invention.

FIG. 29 is a general block diagram illustrating the functionality of aphotographing apparatus as a fourth embodiment of the present invention.

FIG. 30 is a flowchart of a fourth photographing method of the presentinvention.

FIG. 31 is a general block diagram illustrating the functionality of aphotographing apparatus as a fifth embodiment of the present invention.

FIG. 32 is a flowchart of a fifth photographing method of the presentinvention.

FIG. 33 is a general block diagram illustrating the functionality of aphotographing apparatus as a sixth embodiment of the present invention.

FIG. 34 is a flowchart illustrating the measurement of a 3D model usingimages photographed from the air and on the ground.

FIG. 35 is a general diagram illustrating the configuration of a sixthembodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention will be described below with reference to thefigures. Identical or corresponding components in the figures are givenwith the same or similar reference numerals and symbols, and theirdescriptions will not be repeated.

First Embodiment

FIG. 1 is a general block diagram illustrating the functionality of afirst embodiment of the present invention. FIG. 3 shows the systemconfiguration of the first embodiment of the present invention. Anobject 10 is a tangible substance such as a measuring object ormanufacturing object, and may be, for example, a structure of variouskinds such as an architecture included in the field of city planning,construction, maintenance or a cultural property, a person, or alandscape. In the figure, a model forming apparatus of the presentinvention includes a photographing unit housing 100, a photographingsection 110, photographing position measurement sections 120, 125 forobtaining information on the photographing position of the photographingsection 110, an image data storage section 130 for storing plural imagedata of the object 10 with known positional points, a display section140 such as a liquid crystal display or CRT, and a computationprocessing section 200. The model forming apparatus may be called, forexample, a virtual guiding system (a photographing guiding sysytem).

The photographing unit housing 100 integrally holds the photographingsection 110 and the photographing position measurement sections 120,125. The photographing unit housing 100 is made small and lightweightfor easy operation by a photographer. As the photographing section 110,for example, a digital camera for civilian use, preferably one capableof video output may be used. The number of pixels may be selecteddepending on the resolution required. As the photographing positionmeasurement section 120, a GPS is used so that it can measure thephotographing position (X, Y, Z) of the photographing section 110 asterrestrial latitude, longitude and altitude. Preferably, thephotographing position measurement section 120 can switch between thekinematic and real-time kinematic modes. The photographic positionmeasurement section 125 uses a 3-axis angle sensor for measuring thephotographing attitude (Yaw, Pitch, Law) of the photographing section110. The photographing position measurement section 125 has a resolvingpower of 0.05°, for example, with a static accuracy of 1° RMS and adynamic accuracy of 3° RMS.

The image data storage section 130 is adapted to use a medium forstoring image information, such as a flexible disk, a MD or a DVD. A keyentry section 132 is used by an operator to make key entry into thecomputation processing section 200, and includes, for example, akeyboard, a mouse or a touch panel. The display section 140 has a mode Iwith a 3-divided display area of a monitor image display section 142, amodel display section 144 and a measurement condition display section(photographing condition display section) 146, and a mode II with a3-divided display area of the measurement condition display section 146,a photographing position entry screen 148 and a display screen forobject model 149 as viewed from photographing position. The mode I issuitable to allow recognizing an unmodeled part of the object 10. Themode II is suitable to display photographing instruction informationrelated to photographing the unmodeled part of the object 10 so as toallow taking a ground-based photograph as a supplement to a basic aerialphotograph.

The computation processing section 200 may be implemented by, forexample, a general-purpose laptop computer, and receives the imageinformation obtained by photographing with the photographing section110, and the position information (X, Y, Z) and the attitude information(Yaw, Pitch, Law) on the photographing section 110 obtained bymeasurement with the photographing position measurement sections 120,125. The general-purpose laptop computer includes an electromagneticstorage device such as a flexible disk storage device or CD-ROM, whichis used as the image data storage section 130. The general-purposelaptop computer stores as software a recognition section 150, a modelforming section 155, a photographing instruction information section160, a measurement setting section 170, a photographing conditioncalculation section 180 and a reference point display position computingsection 190.

The recognition section 150 recognizes an unmodeled part of the object10 based on three-dimensional model data stored in the image datastorage section 130. The model forming section 155 uses the image datastored in the image data storage section 130 to form a three-dimensionalmodel of the object 10. The calculation procedures to generate athree-dimensional model are specifically described in JP-A-2002-352224,JP-A-2003-42730 and JP-A-2003-65737 proposed by the present applicantand so on. The model forming section 155 may be a 3D measurement systemPI-3000V2 (trade name) available from TOPCON Corporation. The modelforming section 155 can measure in 3D and model all in one, from aerialphotographs to digital camera images. The recognition section 150extracts an area of the object 10 where the model forming section 155cannot form a three-dimensional model, and specifically recognizes theorientation of a surface of the object 10 where no or insufficient imageinformation is available.

The photographing instruction information section 160 obtainsphotographing instruction information related to photographing theunmodeled part of the object 10 recognized by the recognition section150. The photographing instruction information obtained with thephotographing instruction information section 160 is displayed, forexample, on the measurement condition display section 146, thephotographing position entry screen 148 and the display screen forobject model 149 as viewed from photographing position. Thephotographing instruction information includes geodesic positioninformation for measuring the object 10, the distance between the object10 and the photographing section 110, the focal length of thephotographing section 110, and the like. Preferably, the measurementcondition display section 146 graphically displays the photographinginstruction information calculated by the photographing instructioninformation section 160 as superimposed over a plan view of an areacontaining, or over a stereo model of, the object 10.

The measurement setting section 170 sets measurement conditionsnecessary to form a three-dimensional model of the object 10. Themeasurement conditions can be derived from the theory of stereo imagemeasurement, and include, for example, the measurement accuracy, and thebaseline lengths of, and the overlap (redundancy ratio) between the leftand right stereo photographs. The photographing condition calculationsection 180 obtains photographing conditions that satisfy themeasurement conditions set by the measurement setting section 170. Forexample, the photographing condition calculation section 180 obtains,based on the most recent image photographed by the photographing section110, photographing conditions for the photographing section 110 for thenext photographing. The photographing conditions may preferably beconsistent with the photographing instruction information obtained bythe photographing instruction information section 160.

The reference point display position computing section 190 displays inreal time a reference point as superimposed over the object 10 on afinder image being displayed in the monitor image display section 142.This superimposed state is calculated by the reference point displayposition computing section 190, which checks image information from thefinder of the photographing section 110 to calculate whether or not thecoordinate of the reference point entered beforehand falls within thedisplayable area in the finder image.

FIG. 4 is an exemplary screen view displayed on a liquid crystal displaypanel, which is used as the monitor image display section 142, the modeldisplay section 144 and the measurement condition display section 146.The monitor image display section 142 displays, on the liquid crystaldisplay panel, the finder image of the photographing section 110, whichhas been sent to the computer constituting the computation processingsection 200 via an interface of various types. Preferably, the referencepoint relating to the object 10 contained in the finder image may alsobe displayed as being superimposed, by the reference point displayposition computing section 190. This allows determining how manyreference points are being contained in the monitor image displaysection 142 when the photographing section 110 performs photographing.The monitor image display section 142 may be configured to call up andto display first and second photographed images sequentially, which havebeen photographed by the photographing section 110 and stored in theimage data storage section 130.

The apparatus as shown in FIG. 4 is designed to display three screens,of which two from the left show images previously photographed andstored in the image data storage section 130, and the rightmost oneshows an image currently being captured by the photographing section110. The photographer can easily find the photographing point whilechecking in real time the three screens being displayed in the monitorimage display section 142. Also, previously photographed images can bechecked by having images stored in the image data storage section 130sequentially displayed in all of the three screens.

The model display section 144 can display results modeled for the object10 by the model forming section 155 as viewed with the photographingposition and posture of the photographing section 110. This allowsdetermination of the photographing position for the object 10 whilerecognizing the missing part in the model of the object 10 and checkingan appropriate supplemental photographing direction for the object 10.The object 10 can be displayed three-dimensionally using, for example,an OpenGL function. The object 10 is measured three-dimensionally, andits reconstructed image can be displayed through wireframes ortexture-mapping. This allows an image of the object 10 to be displayedin the model display section 144 while changing the viewpoint andresolution in real time. The unmodeled part of the object 10 recognizedby the recognition section 150 is not displayed in the object 10 imagedisplayed on the model display section 144 as it is a missing surface,which allows the photographer to recognize that supplementalphotographing is necessary.

The measurement condition display section 146 displays the positioninformation (X, Y, Z) of the photographing section 110 obtained bymeasurement with the photographing position measurement section 120, andthe attitude information (Yaw, Pitch, Law) of the photographing section110 obtained by measurement with the photographing position measurementsection 125. The measurement condition display section 146 also displaysthe measurement conditions set by the measurement setting section 170,such as the baseline length (B), distance (H), focal length (f), pixelresolving power (p), accuracy (Δ XY, Δ Z), overlap (redundancy ratio)and so on.

The image data storage section 130 stores images photographed by thephotographing section 110 and the photographing positions at which theimages have been photographed. The photographing condition calculationsection 180 calculates the overlap ratio of the finder image (secondphotographed image) displayed in the monitor image display section 142with respect to the last photographed image (first photographed image)stored in the image data storage section 130. Using the measurementaccuracy, baseline, photographing position, photographing angle andinterval between the photographing sections for three-dimensionalmeasurement of the object 10 set in the measurement setting section 170,the photographing condition calculation section 180 calculates anapproximate accuracy of the object 10, and outputs it to the measurementcondition display section 146 for display.

The photographing condition calculation section 180 obtains anapproximate accuracy by the following equations:ΔXY=H*δp/f  (1)ΔZ=H*H*δp/(f*B)  (2)Here, δp represents the pixel resolving power of the photographingsection 110 or the reading resolving power of the scanner, f representsthe focal length, which are known according to the type of thephotographing section 110 used. The baseline length B is calculated fromthe position measured by the photographing position measurement section120 where an image has been photographed, and the position measured bythe photographing position measurement section 120 where the nextphotographing is performed. The photographing distance H can becalculated, if any reference point or a 3D model is present in the areato be photographed, based on the position of the photographing positionmeasurement section 120 and that of the reference point or 3D model. Ifany reference point or a 3D model is not present in the photograph, anapproximate value should be entered. From the parameters used in theequations above, namely the resolving power δp, focal length f, baselinelength B and photographing distance H, approximate accuracies Δ XY, Δ Zafter measurement of a stereo model just about to be photographed can becalculated.

FIG. 5 is another exemplary screen view displayed on the liquid crystaldisplay panel, which is used as the measurement condition displaysection 146, the photographing position entry screen 148 and the displayscreen for object model 149 as viewed from photographing position. Thephotographing instruction information section 160 obtains photographinginstruction information related to photographing an unmodeled part ofthe object 10 recognized by the recognition section 150. Thus, bydesignating an appropriate photographing position on the photographingposition entry screen 148 using the entry section 132, the displayscreen for object model 149 displays a model image of the object asviewed from the photographing position.

FIG. 6 illustrates an exemplary photographing position entry screen usedto enter the photographing position. Preferably, the photographingposition entry screen 148 may use, for example, a plan view of theobject displayed on the liquid crystal display panel. On thephotographing position entry screen 148, the entry section 132 such as alight pen or a mouse is used to enter photographing positions 1-6. Then,the model forming section 155 forms 3D models of the object as viewedfrom the positions entered as photographing positions, and displaysvisual images on the display screen for object model 149 as viewed fromphotographing position. The display section 140 may display thesephotographing positions as stereo photographing positions, together withan area desired to be modeled on a plan view as shown in FIG. 7, or on amodel image as shown in FIGS. 8 and 9, so as to guide the photographer.

FIG. 10 shows exemplary visual images of the object as viewed from thedesignated photographing positions displayed on the display screen forobject model 149. In the figure, reference numerals 1-6 correspond tothe designated photographing positions 1-6 of FIG. 6. When designatedphotographing positions 1-6 are entered on the photographing positionentry screen 148, 3D images of the object as viewed from the designatedphotographing positions 1-6 and at an angle toward the photographingsection 110 are formed and displayed on the display screen for objectmodel 149.

The photographing work with the apparatus configured as described abovewill be described. FIG. 11 is a flowchart illustrating the measurementto form a 3D model using images photographed from the air and on theground. Aerial photographing is first performed (S300). Theaerial-photographed image may be of various kinds, such as aerialphotographs (for example provided by the Geographical Survey Institute),those obtained with a photographing system on a helicopter, airship,balloon, etc., and those image data captured using a paraglider with anengine. Note that the internal orientation elements under thephotographing conditions of the photographing section 110 need to beknown.

Now, on the ground, reference points necessary to generate a 3D modelare obtained with a measuring instrument or a GPS (S310). In this event,measurement values with a measuring instrument should be represented interms of a global positioning coordinate system via a GPS foruniformity. Using the aerial-photographed image obtained in S300, themodel forming section 155 performs a measurement for forming a 3D modelof the object 10 (S320). In this event, the reference points obtained inS310 may be used as appropriate, and three-dimensional model datagenerated by the model forming section 155 based on theaerial-photographed image are stored in the image data storage section130.

The recognition section 150 recognizes an unmodeled part of the object10 based on the three-dimensional model data stored in the image datastorage section 130 (S330). Using a virtual guiding system, thephotographing section 110 performs ground-based photographing (S340). Inthis event, the photographing section 110 may preferably photograph theunmodeled part of the object 10 recognized by the recognition section150 using photographing instruction information obtained by thephotographing instruction information section 160. Before this step, thereference points measured in S310 and the 3D model data generated inS320 should be stored beforehand in the virtual guiding system. Thisallows the monitor image display section 142 to display in real time anobject image at an angle oriented by the photographing section 110, withthe reference points obtained by the measurement in S310 displayed assuperimposed over the object image, while the photographing positionsare confirmed. The model display section 144 displays the 3D modelgenerated in S320 at an angle as viewed through the photographingsection 110. The measurement condition display section 146 allowsconfirmation of the measurement accuracy and/or the overlapping staterelating to the monitor image display section 142. These functions allowthe photographer to bring the photographing section 110 to a mostsuitable photographing position where a post process such as 3D modelgeneration is in view.

The model forming section 155 performs a stereo measurement on the imagephotographed using the virtual guiding system, to generate aground-based 3D model (S340). It is determined based on the 3D modelgenerated in S340 whether or not supplemental photographing of theobject 10 is necessary (S350). The determination of the necessity forsupplemental photographing may preferably be based on the recognition ofan unmodeled part by the recognition section 150, and should take intoaccount the difference in resolution between the aerial-photographedimage and the ground-photographed image. When supplemental photographingof the object 10 is necessary, the process returns to S330. On the otherhand, when supplemental photographing of the object 10 is not necessary,the process proceeds to S360.

The model forming section 155 generates an overall 3D model (S360). Atthis time, the model forming section 155 performs simultaneously bundleadjustments on the pass points and the tie points in each stereo modelimage photographed from the air or on the ground (S370), to therebyuniform the coordinate systems between the stereo model imagesphotographed from the air or on the ground and hence to generate anoverall 3D model. Note that the virtual guiding system can automaticallyform a ground-based 3D model and generate an overall 3D model for eachstereo model image photographed from the air or on the ground, whileperforming bundle adjustments. According to the flowchart of FIG. 11,S370 is performed after S360. However, S360 and S370 may be performedsimultaneously.

Now, a description will be made of an example of 3D model generationbased on aerial photographs, air images and ground-based images usingthe foregoing virtual guiding system according to the present invention.FIG. 12 shows an overall site plan of the object for the 3D modelgeneration using the virtual guiding system according to the presentinvention. The object is an 8-story building owned by TOPCONCorporation, located at the place of registration of the main office ofTOPCON Corporation. The used images include an aerial photograph (FIG.13) as basic image information on the overall object, imagesphotographed by the photographing section 110 on a powered glider (FIG.15) to model the object from the air, and those of ground-level sidesurfaces of the object (FIG. 18) obtained with the photographing section110 of the virtual guiding system. These three types of images aremerged and formed into a 3D model by the model forming section 155. Thereference points are obtained by measuring the side surfaces of thebuilding as an object with a measuring instrument, and converting theirpositions measured by the measuring instrument into GPS coordinatesusing GPS-measured points within the site of the object.

The respective processes of photographing and analysis will be describedbelow.

(1) Base Generation by Aerial Photograph Analysis

FIG. 13 is an aerial photograph of about 4 km square around ItabashiWard, Tokyo, including the location where the main office of TOPCONCorporation is registered, provided by the Geographical SurveyInstitute. The camera for photographing may be a general one for aerialphotographing purposes. The focal length, view angle, etc., may beselected as required for photographing.

FIG. 14 shows a modeled image of an object area for 3D model generationby the model forming section. In order to use the aerial photograph as abase (foundation) for the analysis of the 3D model generation objectarea, the film is scanned with a scanner at a resolution of, forexample, 600 dpi, and the scanned data is subjected to an interiororientation by the model forming section 155. It should be appreciatedthat stereo matching is not performed but a simple way of forming a 3Dimage is used in FIG. 14. However, if the object includes ups and downssuch as mountains, a part of the 3D model generation object areaincluding the ups and downs may preferably be subjected to a stereomatching process to form a 3D model.

(2) 3D Model Generation Using Aerial-Photographed (Powered Glider)Images

FIG. 15 show images photographed with the photographing section 110 on apowered glider to model the object from the air. FIG. 16 illustratesphotographing from a powered paraglider (paraglider with an engine).Different from airplanes, powered paragliders are not subject torestrictions regarding altitude, etc., and provide inexpensive and safephotographing. Aside from powered gliders, helicopters, balloons, lightairplanes that allow large-scale aerial photographing, and the like mayalso be used. Digital cameras for civilian use may be used as the camerafor photographing.

FIG. 17 illustrates an exemplary process of forming a 3D model using theaerial photograph. The procedures for analysis to form a 3D model are asfollows. The model forming section 155 determines the orientation usingthe reference points obtained by measurement with a measuringinstrument, and performs a polyline measurement on the edges and thefront yard of the building as an object (FIG. 17 a). Here, the polylinemeasurement refers to a three-dimensional measurement performed manuallyor through automatic position detection, in which corresponding pointsare designated manually or measured semi-automatically using acorrelation process. The wall surfaces of the building as an object areautomatically measured, and meshes are generated for its flat portions(FIG. 17 b). The external shape of the building as an object isdisplayed (FIG. 17 c). Preferably, the building as an object may bedisplayed with texture attached, as in a second embodiment, giving it anenhanced feel. The external shape of the building that has been formedinto a 3D model is integrated with the aerial photograph to be mergedwith general image information on the area around the object (FIG. 17d). At this time, a part of the wall surfaces of the building as anobject has not been photographed because of the restrictions related toaerial photographing, and therefore is missing in the 3D model.

(3) Addition of Ground-Based 3D Model from Virtual Guiding System

FIG. 18 shows the scene of photographing the object with thephotographing section 110 from an appropriate supplemental direction,and additional images photographed. In this event, the display screen ofthe virtual guiding system is as shown in FIG. 4, for example. Themonitor image display section 142 displays an image of the object asviewed from an appropriate supplemental photographing direction.Although a part of the wall surfaces of the building as an object hasnot been photographed because of the restrictions related to aerialphotographing, photographing from an appropriate supplemental directionallows obtaining the missing wall surface image information with precisereference point information.

Second Embodiment

FIG. 19 is a general block diagram illustrating the functionality of asecond embodiment of the present invention. Components in FIG. 19 havingthe same functions as those in FIG. 1 are given the same referencenumerals, and their descriptions will not be repeated. As shown in thefigure, the image data storage section 130 stores three-dimensionalmodel data and object image data of the object 10. Here, thethree-dimensional model data refer to data in which positional data ofpoints of the object have been obtained, for example with a measuringinstrument or a laser scanner. The object image data refer to mere imagedata in which no positional data of points have been obtained. The modelforming section 155 forms the three-dimensional model data of the object10 using the object image data. A texture attaching section 192 dividesthe three-dimensional model data of the object 10 into plural areas, andattaches texture of the object image data to each divided area based onthe relationship between each divided area and the photographingdirection.

FIG. 20 is a flowchart illustrating the second embodiment of the presentinvention. The model forming section 155 obtains model data of theobject 10, using the three-dimensional model data of the object 10stored in the image data storage section 130 and as necessary the objectimage data obtained by photographing with the photographing section 110(S210). The model dislay section 144 displays a three-dimensional modelbased on the three-dimensional model data of the object 10 (S320).

The recognition section 150 recognizes an unmodeled part of the object10 based on the three-dimensional model data of the object 10 (S230).The photographing instruction information section 160 obtainsphotographing instruction information related to photographing theunmodeled part (S240). The display section 140 displays thethree-dimensional model and the photographing instruction informationtogether (S250). That is, the model display section 144 displays thethree-dimensional model while the measurement condition display section146 displays the photographing instruction information. The object 10 isphotographed with the photographing section 110 in accordance with thethree-dimensional model and the photographing instruction informationdisplayed in the display section 140 by the photographer (S260).

The model forming section 155 generates an overall 3D model (S270). Atthis time, the model forming section 155 performs simultaneously bundleadjustments on the pass points and the tie points in each stereo modelimage photographed from the air or on the ground (S280), to therebyuniform the coordinate systems between the stereo model imagesphotographed from the air or on the ground and hence to generate anoverall 3D model. That is, according to the flowchart of FIG. 20, S280is performed after S270. However, S270 and S280 may be performedsimultaneously. The texture attaching section 192 attaches texture ofthe object image data to each divided area of the three-dimensionalmodel data of the object 10 based on the relationship between eachdivided area and the photographing direction (S290). A three-dimensionalmodel of the object is displayed using the object image data withtexture attached (S292).

FIG. 21 shows an image of the object in which wireframes are attached tothe aerial photograph. FIG. 22 shows an image of the object in whichtexture is attached to the aerial photograph. Here, the texture and thewireframes are obtained from the 3D model obtained by the model formingsection 155 from the ground-based images supplemented by the virtualguiding system. The texture is a term used in the graphics field and soon, and refers to what is drawn on a surface of a figure to express apattern and a feel of a substance. The texture is used to give astereoscopic effect to a two-dimensional image of an object. Thewireframes compose a diagram of polygonal line segments, for exampleconnecting the vertexes of a multitude of polygons, such as triangles,representing the surface shape of an object. A part of the wall surfacesof the building as an object has not been photographed because of therestrictions related to aerial photographing. By photographing theobject from an appropriate supplemental direction with the virtualguiding system and then forming a 3D model with the photographed imageof the object obtained, the entire object can be subjected to a 3Dmodeling process.

FIG. 23 illustrates the 3D model added with a building adjacent to theobject. The building adjacent to the object has been added after the 3Dmodeling process for the entire object as shown in FIG. 22 using theaerial-photographed image. In this manner, additional analysis using thevirtual guiding system or air images or aerial photographs can easilyhandle a part desired to be supplemented or added in the 3D modelingprocess.

Now, a description will be made of a recognition process by therecognition section 150 of recognizing an unmodeled part of the objectbased on the three-dimensional model data. The recognition process bythe recognition section 150 is described in step S120 of FIG. 2 and stepS230 of FIG. 20. Here, an automatic recognition process will bedescribed. Note, however, that an unmodeled part of the object (anarchitecture) may be designated on the display 140 via a user interfacefor calculation.

FIG. 24 is a flowchart of the recognition of a missing part. Therecognition section 150 searches for a part at a higher elevation than abase model (S500). For example, in the first embodiment, FIG. 14corresponds to a 3D model as the base model generated from an aerialphotograph. A 3D model generated by superimposing a building over thebase model is as shown in FIG. 17(d). Since they are 3D models, a partat a higher elevation than the base part corresponds to an architecture.

The recognition section 150 extracts a figure on the basis of theelevation value of the base (S510). That is, the recognition section 150extracts a part with a higher elevation value than that of the base as abuilding based on the search results in S500. As a result, portions (1),(2), (3) and (4) of FIG. 25 are extracted as parts of a building.

The recognition section 150 recognizes a missing part in the base modelgenerated from an aerial photograph. For an architecture, line segmentsconnecting points with a higher elevation value than that of the baseshould inevitably be closed, and therefore any discontinuous portionshould be a missing part. Thus, portions (5), (6) of FIG. 25 arerecognized as missing parts. Although the processes from S500 to S520solely allow recognition of a missing part in the base model generatedfrom an aerial photograph, the next process of S530 may further beperformed for more reliability.

The recognition section 150 checks whether or not there is any adjacentsurface to the part with a higher elevation value than the baseextracted in S510 (S530). In this event, since the sides of thearchitecture are surfaces, the portions (1)-(4) of FIG. 25 are checkedas to whether or not there is any such surface. FIG. 26 is a 3D modelview illustrating the missing parts as recognized through thestereoscopic shape. The process of S530 allows recognition of missingareas corresponding to the portions (5), (6) of FIG. 25. One exemplaryrecognition process has been described above. Note, however, that therecognition section 150 may recognize a missing part in the base modelgenerated from an aerial photograph in various processes.

Now, the process of obtaining photographing information related tophotographing the unmodeled part will be described. The process ofobtaining photographing information is described in step S130 of FIG. 2and step S240 of FIG. 20. In the generation of a 3D model, for example,when the required accuracy (δxy, δz) for modeling an architecture hasbeen entered, the photographing distance of cameras H and the distancebetween the cameras B can be calculated by the following equations (3)and (4):H=δxy·f/δp  (3)B=H·H·δp/(f·δz)  (4)where the focal length of the cameras f and the pixel resolving power δpare known. The photographing position can be obtained as shown in FIG.27 by determining the H and B along the line normal to the missing part.

Third Embodiment

FIG. 28 is a general diagram illustrating the constitution of a thirdembodiment of the present invention. In the first embodiment, a digitalcamera and a laptop computer are used in combination as shown in FIG. 3.In the present embodiment, however, a GPS, a digital camera and asmall-sized host PC are constituted integrally. The small-sized host PCmay be implemented by, for example, a portable information processingterminal such as a PDA. The functions of the GPS, digital camera andsmall-sized host PC are the same as those described in relation to thefirst embodiment, and their detailed descriptions will not be repeated.

As has been described above, in the bundle adjustment of an aerialphotograph and an image photographed on the ground with thephotographing section 110 of the virtual guiding system, it is importantto supplement image data of the object by photographing from a missingdirection. According to the present embodiment, the recognition section150 can recognize a missing photographing direction for the object, andthe photographing instruction information section 160 can present whatneeds to be noted in supplemental photographing from the missingdirection. This allows efficient 3D model construction from the air andon the ground. Also according to the present embodiment, imagesphotographed with a photographing section 110 of various types and atvarious resolutions can be subjected to a collective bundle adjustment,securing consistency among image measurement data of various kinds.Further, a textured 3D model can be generated using DSM (Digital StereoMatching) data obtained by measurement through high-accuracy stereomatching, and displayed as viewed from any viewpoint and at variousresolutions.

Fourth Embodiment

FIG. 29 is a block diagram illustrating the functionality of aphotographing apparatus as a fourth embodiment of the present invention.The general construction of the apparatus is the same as shown in FIG. 3which has been used to describe the first embodiment. An object 10 is atangible substance such as a measuring object or manufacturing object,and may be, for example, a structure of various kinds such as anarchitecture included in the field of city planning, construction,maintenance or a cultural property, a person, or a landscape. In thefigure, a photographing apparatus of the present invention includes aphotographing unit housing 100, a photographing section 110,photographing position measurement sections 120, 125 for obtaininginformation on the photographing position of the photographing section110, an image data storage section 130 for storing plural image data ofthe object 10 with known positional points, a monitor image displaysection 142, a model display section 144, a measurement conditiondisplay section 146 and a computation processing section 200. Thephotographing apparatus may be called, for example, a virtual guidingsystem as described previously.

The computation processing section 200 may be implemented by, forexample, a general-purpose laptop computer, and receives the imageinformation obtained by photographing with the photographing section110, and the position information (X, Y, Z) and the attitude information(Yaw, Pitch, Law) on the photographing section 110 obtained bymeasurement with the photographing position measurement sections 120,125. The general-purpose laptop computer includes an electromagneticstorage device such as a flexible disk storage device or CD-ROM, whichis used as the image data storage section 130. A liquid crystal displaypanel of the general-purpose laptop computer is used as the monitorimage display section 142, the model display section 144 and themeasurement condition display section 146. The general-purpose laptopcomputer stores as software a model forming section 155, a photographingcondition measurement section (photographing condition setting section)220 and a reference point display position computing section 190.

The model forming section 155 uses the image data stored in the imagedata storage section 130 to form a three-dimensional model of the object10. The calculation procedures to generate a three-dimensional model arespecifically described in JP-A-2004-037270 proposed by the presentapplicant and so on. The model forming section 155 may be a 3Dmeasurement system PI-3000V2 (trade name) available from TOPCONCorporation. The model forming section 155 can measure in 3D and modelall in one, from aerial photographs to digital camera images.

The photographing condition measurement section 220 measures themeasurement accuracy, baseline, photographing position, photographingangle, and interval between the photographing sections forthree-dimensional measurement on the object 10 from the photographingposition information on first and second photographed images, and themeasurement condition display section 146 displays the measurementresults. The first and second photographed images refer to a firstphotographed image stored in the image data storage section 130 togetherwith its photographing position and a second photographed imagedisplayed in the monitor image display section 142, and are used to forma three-dimensional model of the object 10 in the model forming section155.

The reference point display position computing section 190 checks imageinformation from the finder of the photographing section 110 tocalculate whether or not the coordinate of a reference point enteredbeforehand falls within the displayable area in the finder image, anddisplays in real time the reference point as superimposed over theobject 10 on the finder image being displayed in the monitor imagedisplay section 142.

An exemplary screen view to be displayed on the liquid crystal displaypanel is illustrated in FIG. 4 described previously, which is used asthe monitor image display section 142, the model display section 144 andthe measurement condition display section 146, as has been describedpreviously. The monitor image display section 142 is used also as afinder image display section 290, and displays, on the liquid crystaldisplay panel, the finder image information of the photographing section110, which has been sent to the computer constituting the computationprocessing section 200 via an interface of various types. Preferably,the reference point relating to the object 10 contained in the finderimage may also be displayed as being superimposed, by the referencepoint display position computing section 190. This allows determininghow many reference points are being contained in the monitor imagedisplay section 142 when the photographing section 110 performsphotographing. The monitor image display section 142 may be configuredto call up and to display first and second photographed imagessequentially, which have been photographed by the photographing section110 and stored in the image data storage section 130. The apparatus inFIG. 4 is designed to display three screens, of which two from the leftshow images previously photographed and stored in the image data storagesection 130, and the rightmost one shows an image currently beingcaptured by the photographing section 110. The photographer can easilyfind the photographing point while checking in real time the threescreens being displayed in the monitor image display section 142. Also,previously photographed images can be checked by having images stored inthe image data storage section 130 sequentially displayed in all of thethree screens.

The image data storage section 130 stores images photographed previouslyby the photographing section 110 and the photographing positions atwhich the images have been photographed. The photographing conditionmeasurement section 220 calculates the overlap ratio of the finder image(second photographed image) displayed in the monitor image displaysection 142 with respect to the last photographed image (firstphotographed image) stored in the image data storage section 130. Usingthe measurement accuracy, baseline, photographing position,photographing angle and interval between the photographing sections forthree-dimensional measurement of the object 10 measured, thephotographing condition measurement section 220 calculates anapproximate accuracy of the object 10, and outputs it to the measurementcondition display section 146 for display. The approximate accuracyobtained by the photographing condition measurement section 220 issimilar to that obtained by the foregoing photographing conditioncalculation section 180.

The photographing work with the apparatus configured as described abovewill be described. FIG. 34 is a flowchart illustrating the measurementto form a 3D model using images photographed from the air and on theground. Aerial photographing is first performed (S900). Theaerial-photographed image may be of various kinds, such as aerialphotographs (for example provided by the Geographical Survey Institute),those obtained with a photographing system on a helicopter, airship,balloon, etc., and those image data captured using a paraglider with anengine. Note that the internal orientation elements under thephotographing conditions of the photographing section 110 need to beknown.

Now, on the ground, reference points necessary to generate a 3D modelare obtained with an auto-tracking measuring instrument or a GPS (S910).In this event, measurement values with an auto-tracking measuringinstrument should be represented in terms of a global positioningcoordinate system via a GPS for uniformity. Using theaerial-photographed image obtained in S900, the model forming section155 performs a measurement for forming a 3D model of the object 10(S920). In this event, the reference points obtained in S910 may be usedas necessary.

Using a virtual guiding system, the photographing section 110 performsground-based photographing (S930). Before this step, the referencepoints measured in S910 and the 3D model data generated in S920 shouldbe stored beforehand in the virtual guiding system. This allows themonitor image display section 142 to display in real time an objectimage at an angle toward the photographing section 110, with thereference points obtained by the measurement in S910 displayed assuperimposed over the object image, while the photographing positionsare confirmed. The model display section 144 displays the 3D modelgenerated in S920 at an angle as viewed through the photographingsection 110. The measurement condition display section 146 allowsconfirmation of the measurement accuracy and/or the overlapping staterelating to the monitor image display section 142. These functions allowthe photographer to bring the photographing section 110 to a mostsuitable photographing position where a post process such as 3D modelgeneration is in view.

The model forming section 155 performs a stereo measurement on the imagephotographed using the virtual guiding system, to generate aground-based 3D model (S940).

The model forming section 155 generates an overall 3D model (S950). Themodel forming section 155 performs simultaneously bundle adjustments onthe pass points and the tie points in each stereo model imagephotographed from the air or on the ground, to thereby uniform thecoordinate systems between the stereo model images photographed from theair or on the ground and hence to generate an overall 3D model. Notethat the virtual guiding system can automatically form a ground-based 3Dmodel and generate an overall 3D model for each stereo model imagephotographed from the air or on the ground, while performing bundleadjustments.

The 3D model generation based on aerial photographs, air images andground-based images using the foregoing virtual guiding system accordingto the present invention is similar to the 3D model generation describedwith reference to FIGS. 12-18.

Fifth Embodiment

In the fourth embodiment, a digital camera and a laptop computer areused in combination as shown in FIG. 3. However, as described inrelation to the first and second embodiments, a GPS, a digital cameraand a small-sized host PC may be configure integrally. The small-sizedhost PC may be implemented by, for example, a portable informationprocessing terminal such as a PDA. The functions of the GPS, digitalcamera and small-sized host PC are the same as those describedpreviously, and their detailed descriptions will not be repeated. Thegeneral block diagram of the fifth embodiment of the present inventionillustrating the functionality is shown in FIG. 31, and the method ofphotographing is shown in the flowchart of FIG. 32. The generalconstitution of the fifth embodiment of the present invention is shownin FIG. 28, which is common to the third embodiment. Here, thedescription will not be repeated.

Sixth Embodiment

FIG. 35 is a general diagram illustrating the constitution of a sixthembodiment of the present invention. As shown in the figure, aphotographing apparatus of the present invention includes aphotographing unit housing 100, a photographing section 110,photographing position measurement sections 120, 125 for obtaininginformation on the photographing position of the photographing section110, an image data storage section 130, a monitor image display section142, a model display section 165, a measurement condition displaysection 146, a photographing position designating section 280, and acomputation processing section 200. In FIG. 35, components having thesame functions as those in FIG. 3 are given the same reference numerals.The general block diagram of the sixth embodiment of the presentinvention illustrating the functionality is shown in FIG. 33. Here, thedescriptions will not be repeated.

The photographing position designating section 280 allows designation ofthe photographing position of the photographing section 110, and may beimplemented by, for example, a mouse connected to a personal computer,or cursor keys. The model display section 165 displays, for athree-dimensional model of the object 10 formed by the model formingsection 155, a three-dimensional model image of the object 10 as viewedfrom the designated photographing position of the photographing section110 based on image data stored in the image data storage section 130 andthe photographing position designated by the photographing positiondesignating section 280.

An exemplary photographing position entry screen used to enter thephotographing position with the photographing position designatingsection 280 is shown in FIG. 6 described previously. Preferably, thephotographing position entry screen may use, for example, a plan view ofthe object displayed in the model display section 165. On thephotographing position entry screen, the tools such as a light pen or amouse is used to enter photographing positions 1-6. Then, the modelforming section 155 forms 3D models of the object as viewed from thepositions entered as photographing positions, and displays visual imagesin the model display section 165.

Exemplary visual images of the object as viewed from the designatedphotographing positions and displayed in the model display section 165are shown in FIG. 10 described previously. When designated photographingpositions 1-6 are entered with the photographing position designatingsection 280, 3D images of the object as viewed from the designatedphotographing positions 1-6 and at an angle toward the photographingsection 110 are formed and displayed on the monitor image display screen142. The system configuration of the present embodiment may be as shownin FIG. 28.

As has been described above, in the bundle adjustment of an aerialphotograph and an image photographed on the ground with thephotographing section 110 of the virtual guiding system, it is importantto recognize the arrangement of reference points or tie points.According to the present embodiment, the virtual guiding system allowssuch orientation points to be displayed in real time when thephotographing section 110 performs photographing. This allows efficient3D model construction from the air and on the ground. Also according tothe present embodiment, images photographed with a photographing section110 of various types and at various resolutions can be subjected to acollective bundle adjustment, securing consistency among imagemeasurement data of various kinds. Further, a textured 3D model can begenerated using DSM (Digital Stereo Matching) data obtained bymeasurement through high-accuracy stereo matching, and displayed asviewed from any viewpoint and at various resolutions.

The following are the list of reference numerals for major componentsused in the descriptions above.

-   10: measuring object-   110: photographing section-   120: photographing position measurement section-   130: image data storage section-   140: display section-   142: monitor image display section-   144, 165: model display section-   146: measurement condition display section-   150: recognition section-   155: model forming section-   160: photographing instruction information section-   170: measurement setting section-   180: photographing condition calculation section-   190: reference point display position computing section-   220: photographing condition measurement section-   280: photographing position designating section-   290: finder image display section

1. A model forming apparatus comprising: a photographing section forphotographing an object; an image data storage section for storingthree-dimensional model data of the object; a display section fordisplaying a three-dimensional model based on the three-dimensionalmodel data of the object; a recognition section for recognizing anunmodeled part of the object based on the three-dimensional model datastored in the image data storage section; and a photographinginstruction information section for obtaining photographing instructioninformation related to photographing the unmodeled part, wherein thephotographing section photographs the object in accordance with thephotographing instruction information obtained by the photographinginstruction information section.
 2. A model forming apparatuscomprising: a photographing section for photographing an object; animage data storage section for storing at least three-dimensional modeldata of the object or object image data obtained by photographing theobject with the photographing section; a model forming section forobtaining three-dimensional model data of the object from the objectimage data stored in the image data storage section; a display sectionfor displaying a three-dimensional model based on the three-dimensionalmodel data of the object; a recognition section for recognizing anunmodeled part based on the three-dimensional model data stored in theimage data storage section; and a photographing instruction informationsection for obtaining photographing instruction information related tophotographing the unmodeled part, wherein the display section displaysthe three-dimensional model and the photographing instructioninformation together.
 3. The model forming apparatus according to claim2, further comprising: a texture attaching section for dividing themodel formed from the three-dimensional model data by the model formingsection into plural areas and attaching texture of the object image datato each divided area based on the relationship between the each dividedarea and a photographing direction, wherein the display section displaysthe model with texture formed by the texture attaching section.
 4. Themodel forming apparatus according to claim 1, wherein the displaysection is configured to graphically display the photographinginstruction information obtained by the photographing instructioninformation section as superimposed over a plan view of an areacontaining, or over a stereo model of, the object.
 5. The model formingapparatus according to claim 2, wherein the display section isconfigured to graphically display the photographing instructioninformation obtained by the photographing instruction informationsection as superimposed over a plan view of an area containing, or overa stereo model of, the object.
 6. The model forming apparatus accordingto claim 1, further comprising: a measurement setting section forsetting measurement conditions related to photographing the object withthe photographing section; and a photographing condition calculationsection for obtaining photographing conditions satisfying themeasurement conditions set by the measurement setting section.
 7. Themodel forming apparatus according to claim 2, further comprising: ameasurement setting section for setting measurement conditions relatedto photographing the object with the photographing section; and aphotographing condition calculation section for obtaining photographingconditions satisfying the measurement conditions set by the measurementsetting section.
 8. A model forming method, causing a computer toperform the steps of: displaying a three-dimensional model of an objectbased on three-dimensional model data of the object stored in an imagedata storage section; recognizing an unmodeled part of the object basedon the three-dimensional model data; and obtaining photographinginstruction information related to photographing the unmodeled part,wherein a photographing section photographs the object in accordancewith the photographing instruction information.
 9. A model formingmethod, causing a computer to perform the steps of: obtaining model dataof an object from three-dimensional model data of the object stored inan image data storage section or from object image data obtained byphotographing the object with a photographing section; displaying athree-dimensional model based on the three-dimensional model data of theobject; recognizing an unmodeled part of the object based on thethree-dimensional model data of the object; obtaining photographinginstruction information related to photographing the unmodeled part; anddisplaying the three-dimensional model and the photographing instructioninformation together in a display section.
 10. A photographing apparatuscomprising: a photographing section for photographing an object; aphotographing position measurement section for obtaining photographingposition information of the photographing section; an image data storagesection for storing plural image data of the object with a knownpositional point; a model forming section for forming athree-dimensional model of the object using the image data stored in theimage data storage section; and a model display section for displaying,for the three-dimensional model of the object formed by the modelforming section, a three-dimensional model image of the object as viewedfrom a photographing position of the photographing section based on theimage data stored in the image data storage section and thephotographing position information obtained by the photographingposition measurement section.
 11. The photographing apparatus accordingto claim 10, wherein the model forming section forms a three-dimensionalmodel of the object, using a first photographed image stored in theimage data storage section together with the photographing positioninformation and a second photographed image displayed in the modeldisplay section, and the apparatus further comprising: a photographingcondition measurement section for obtaining at least one of measurementaccuracy, baseline, photographing position, photographing angle, andinterval between photographing sections for three-dimensionalmeasurement on the object from the photographing position informationrelated to the first and second photographed images.
 12. A photographingapparatus comprising: a photographing section for photographing anobject; a photographing position designating section for allowingdesignation of a photographing position of the photographing section; animage data storage section for storing plural image data of the objectwith a known positional point; a model forming section for forming athree-dimensional model of the object using the image data stored in theimage data storage section; and a model display section for displaying,for the three-dimensional model of the object formed by the modelforming section, a three-dimensional model image of the object as viewedfrom the designated photographing position of the photographing sectionbased on the image data stored in the image data storage section and thephotographing position designated by the photographing positiondesignating section.
 13. The photographing apparatus according to claim12, wherein the model forming section forming a three-dimensional modelof the object using a first photographed image stored in the image datastorage section together with the photographing position information anda second photographed image displayed in the model display section, andthe apparatus further comprising: a photographing position measurementsection for obtaining photographing position information of thephotographing section; and a photographing condition measurement sectionfor obtaining at least one of measurement accuracy, baseline,photographing position, photographing angle, and interval betweenphotographing sections for three-dimensional measurement on the object,from the photographing position information related to the first andsecond photographed images.
 14. The photographing apparatus according toclaim 10, wherein the image data storage section stores at least a setof stereo images photographed with a photographing device other than thephotographing section that can be used to form a three-dimensional modelof the object in the model forming section.
 15. The photographingapparatus according to claim 12, wherein the image data storage sectionstores at least a set of stereo images photographed with a photographingdevice other than the photographing section that can be used to form athree-dimensional model of the object in the model forming section. 16.The photographing apparatus according to claim 10, wherein the imagedata stored in the image data storage section include at least one of astereo aerial photograph, a stereo small-scale photograph, a stereolow-resolution image and a wide-area photographed image, photographedwith a photographing device other than the photographing section. 17.The photographing apparatus according to claim 12, wherein the imagedata stored in the image data storage section include at least one of astereo aerial photograph, a stereo small-scale photograph, a stereolow-resolution image and a wide-area photographed image, photographedwith a photographing device other than the photographing section. 18.The photographing apparatus according to claim 10, wherein the imagedata obtained by photographing with the photographing section are storedin the image data storage section such that the known positional pointof the object can be displayed being superimposed over the object image;and wherein the object image stored in the image data storage sectioncan therefore be used to form a three-dimensional model of the object inthe model forming section.
 19. The photographing apparatus according toclaim 12, wherein the image data obtained by photographing with thephotographing section are stored in the image data storage section suchthat the known positional point of the object can be displayed beingsuperimposed over the object image; and wherein the object image storedin the image data storage section can therefore be used to form athree-dimensional model of the object in the model forming section. 20.The photographing apparatus according to claim 10, wherein thephotographing section photographs a first photographed image to bestored in the image data storage section together with the photographingposition information and a second photographed image photographed withhaving an overlapping area with the first photographed image withrespect to the object; and wherein the image data storage section isconfigured to sequentially store the first photographed image and thesecond photographed image.
 21. The photographing apparatus according toclaim 12, wherein the photographing section photographs a firstphotographed image to be stored in the image data storage sectiontogether with the photographing position information and a secondphotographed image photographed with having an overlapping area with thefirst photographed image with respect to the object; and wherein theimage data storage section is configured to sequentially store the firstphotographed image and the second photographed image.
 22. Thephotographing apparatus according to claim 10, wherein the photographingsection photographs the object within view sequentially from differentphotographing positions, and the apparatus further comprising: a monitorimage display section for displaying an object image based on image dataof the object obtained by sequentially photographing the object.
 23. Thephotographing apparatus according to claim 12, wherein the photographingsection photographs the object within view sequentially from differentphotographing positions, and the apparatus further comprising: a monitorimage display section for displaying an object image based on image dataof the object obtained by sequentially photographing the object.
 24. Thephotographing apparatus according to claim 22, further comprising areference point display position computing section for displaying, inthe monitor image display section, at least one of a reference point ora pass point superimposed over the image data obtained by sequentiallyphotographing the object.
 25. The photographing apparatus according toclaim 23, further comprising a reference point display positioncomputing section for displaying, in the monitor image display section,at least one of a reference point or a pass point superimposed over theimage data obtained by sequentially photographing the object.
 26. Thephotographing apparatus according to claim 11, further comprising ameasurement condition display section for displaying photographingconditions obtained by measurement with the photographing conditionmeasurement section.
 27. The photographing apparatus according to claim13, further comprising a measurement condition display section fordisplaying photographing conditions obtained by measurement with thephotographing condition measurement section.
 28. A photographingapparatus comprising: a photographing section for photographing anobject; a finder image display section for displaying the object imagebeing photographed by the photographing section; an image data storagesection for storing plural image data of the object with a knownpositional point; and a data forming section for forming reference pointdata or a three-dimensional model of the object using the image datastored in the image data storage section, wherein the finder imagedisplay section is configured to display the reference point data or thethree-dimensional model over the object image.
 29. A photographingmethod, causing a computer to perform the steps of: obtainingphotographing position information for photographing an object with aphotographing section; forming a three-dimensional model of the objectusing image data stored in an image data storage section for storingplural image data of the object with a known positional point; anddisplaying, for the three-dimensional model of the object, athree-dimensional model image of the object as viewed from aphotographing position of the photographing section based on the imagedata stored in the image data storage section and the photographingposition information.
 30. A photographing method, causing a computer toperform the steps of: entering information on a photographing positiondesignated for photographing an object with a photographing section;forming a three-dimensional model of the object using image data storedin an image data storage section for storing plural image data of theobject with a known positional point; and displaying, for thethree-dimensional model of the object, a three-dimensional model imageof the object as viewed from the designated photographing position ofthe photographing section based on the image data stored in the imagedata storage section and the designated photographing position.